Sensitivity Analysis

Sensitivity analysis (SA), in particular global sensitivity analysis (GSA), is now regarded as a discipline coming of age, primarily for understanding and quantifying how model results and associated inferences depend on its parameters and assumptions. Indeed, GSA is seen as a key part of good modelling practice. However, inappropriate SA, such as insufficient convergence of sensitivity metrics, can lead to untrustworthy results and associated inferences. Good practice SA should also consider the robustness of results and inferences to choices in methods and assumptions relating to the procedure. Moreover, computationally expensive models are common in various fields including environmental domains, where model runtimes are long due to the nature of the model itself, and/or software platform and legacy issues. To extract using GSA the most accurate information from a computationally expensive model, there may be a need for increased computational efficiency. Primary considerations here are sampling methods that provide efficient but adequate coverage of parameter space and estimation algorithms for sensitivity indices that are computationally efficient. An essential aspect in the procedure is adopting methods that monitor and assess the convergence of sensitivity metrics. The thesis reviews the different categories of GSA methods, and then it lays out the various factors and choices therein that can impact the robustness of a GSA exercise. It argues that the overall level of assurance, or practical trustworthiness, of results obtained is engendered from consideration of robustness with respect to the individual choices made for each impact factor. Such consideration would minimally involve transparent justification of individual choices made in the GSA exercise but, wherever feasible, include assessment of the impacts on results of plausible alternative choices. Satisfactory convergence plays a key role in contributing to the level of assurance, and hence the ultimate effectiveness of the GSA can be enhanced if choices are made to achieve that convergence. The thesis examines several of these impact factors, primary ones being the GSA method/estimator, the sampling method, and the convergence monitoring method, the latter being essential for ensuring robustness. The motivation of the thesis is to gain a further understanding and quantitative appreciation of elements that shape and guide the results and computational efficiency of a GSA exercise. This is undertaken through comparative analysis of estimators of GSA sensitivity measures, sampling methods and error estimation of sensitivity metrics in various settings using well-established test functions. Although quasi-Monte Carlo Sobol' sampling can be a good choice computationally, it has error spike issues which are addressed here through a new Column Shift resampling method. We also explore an Active Subspace based GSA method, which is demonstrated to be more informative and computationally efficient than those based on the variance-based Sobol' method. Given that GSA can be computationally demanding, the thesis aims to explore ways that GSA can be more computationally efficient by: addressing how convergence can be monitored and assessed; analysing and improving sampling methods that provide a high convergence rate with low error in sensitivity measures; and analysing and comparing GSA methods, including their algorithm settings

A comparison of global sensitivity techniques andsampling method

Inspired by Tarantola et al. (2012), we extend their analysis to include the Latin hypercube and Random sampling methods. In their paper, they compared Sobol’ quasi-Monte Carlo and Latin supercube sampling methods by using a V-function and variance-based sensitivity analysis. In our case we compare the convergence rate and average error between Sobol’, Latin hypercube, and Random sampling methods from the Chaospy library, keeping everything else the same as in their paper. We added the Random sampling method to test if the other two sampling methods are indeed superior. The results from our code confirm the results of their paper, where Sobol’ has better performance than Latin hypercube sampling in most cases, whilst they both have higher efficiency than is achieved with Random sampling. In addition we compared the explicit forms of ‘Jansen 1999’ total effects estimator used in Tarantola et al. (2012) with the ‘Sobol’ 2007’ estimator, again keeping sample sizes and the test function the same. Results confirm that the ‘Jansen 1999’ estimator is more efficient than ‘Sobol’ 2007’. The presentation will also include the Morris sampling method and other test functions to further test efficiency among all the sampling methods on different cases

The Future of Sensitivity Analysis: An essential discipline for systems modeling and policy support

Sensitivity analysis (SA) is en route to becoming an integral part of mathematical modeling. The tremendous potential benefits of SA are, however, yet to be fully realized, both for advancing mechanistic and data-driven modeling of human and natural systems, and in support of decision making. In this perspective paper, a multidisciplinary group of researchers and practitioners revisit the current status of SA, and outline research challenges in regard to both theoretical frameworks and their applications to solve real-world problems. Six areas are discussed that warrant further attention, including (1) structuring and standardizing SA as a discipline, (2) realizing the untapped potential of SA for systems modeling, (3) addressing the computational burden of SA, (4) progressing SA in the context of machine learning, (5) clarifying the relationship and role of SA to uncertainty quantification, and (6) evolving the use of SA in support of decision making. An outlook for the future of SA is provided that underlines how SA must underpin a wide variety of activities to better serve science and society.John Jakeman’s work was supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research, Scientific Discovery through Advanced Computing (SciDAC) program. Joseph Guillaume received funding from an Australian Research Council Discovery Early Career Award (project no. DE190100317). Arnald Puy worked on this paper on a Marie Sklodowska-Curie Global Fellowship, grant number 792178. Takuya Iwanaga is supported through an Australian Government Research Training Program (AGRTP) Scholarship and the ANU Hilda-John Endowment Fun

The future of sensitivity analysis: an essential discipline for systems modeling and policy support

Sensitivity analysis (SA) is en route to becoming an integral part of mathematical modeling. The tremendous potential benefits of SA are, however, yet to be fully realized, both for advancing mechanistic and data-driven modeling of human and natural systems, and in support of decision making. In this perspective paper, a multidisciplinary group of researchers and practitioners revisit the current status of SA, and outline research challenges in regard to both theoretical frameworks and their applications to solve real-world problems. Six areas are discussed that warrant further attention, including (1) structuring and standardizing SA as a discipline, (2) realizing the untapped potential of SA for systems modeling, (3) addressing the computational burden of SA, (4) progressing SA in the context of machine learning, (5) clarifying the relationship and role of SA to uncertainty quantification, and (6) evolving the use of SA in support of decision making. An outlook for the future of SA is provided that underlines how SA must underpin a wide variety of activities to better serve science and society

Investigation of determinism-related issues in the Sobol′ low-discrepancy sequence for producing sound global sensitivity analysis indices

A computationally efficient and robust sampling scheme can support a sensitivity analysis of models to discover their behaviour through Quasi Monte Carlo approximation. This is especially useful for complex models, as often occur in environmental domains when model runtime can be prohibitive. The Sobol' sequence is one of the most used quasi-random low-discrepancy sequences as it can explore the parameter space significantly more evenly than pseudo-random sequences. The built-in determinism of the Sobol' sequence assists in achieving this attractive property. However, the Sobol' sequence tends to deteriorate in the sense that the estimated errors are distributed inconsistently across model parameters as the dimensions of a model increase. By testing multiple Sobol' sequence implementations, it is clear that the deterministic nature of the Sobol' sequence occasionally introduces relatively large errors in sensitivity indices produced by well-known global sensitivity analysis methods, and that the errors do not diminish by averaging through multiple replications. Problematic sensitivity indices may mistakenly guide modellers to make type I and II errors in trying to identify sensitive parameters, and this will potentially impact model reduction attempts based on these sensitivity measurements. This work investigates the cause of the Sobol' sequence's determinism-related issues. References I. A. Antonov and V. M. Saleev. An economic method of computing LPτ-sequences. USSR Comput. Math. Math. Phys. 19.1 (1979), pp. 252–256. doi: 10.1016/0041-5553(79)90085-5 P. Bratley and B. L. Fox. Algorithm 659: Implementing Sobol’s quasirandom sequence generator. ACM Trans. Math. Soft. 14.1 (1988), pp. 88–100. doi: 10.1145/42288.214372 J. Feinberg and H. P. Langtangen. Chaospy: An open source tool for designing methods of uncertainty quantification. J. Comput. Sci. 11 (2015), pp. 46–57. doi: 10.1016/j.jocs.2015.08.008 on p. C90). S. Joe and F. Y. Kuo. Constructing Sobol sequences with better two-dimensional projections. SIAM J. Sci. Comput. 30.5 (2008), pp. 2635–2654. doi: 10.1137/070709359 S. Joe and F. Y. Kuo. Remark on algorithm 659: Implementing Sobol’s quasirandom sequence generator. ACM Trans. Math. Soft. 29.1 (2003), pp. 49–57. doi: 10.1145/641876.641879 W. J. Morokoff and R. E. Caflisch. Quasi-random sequences and their discrepancies. SIAM J. Sci. Comput. 15.6 (1994), pp. 1251–1279. doi: 10.1137/0915077 X. Sun, B. Croke, S. Roberts, and A. Jakeman. Comparing methods of randomizing Sobol’ sequences for improving uncertainty of metrics in variance-based global sensitivity estimation. Reliab. Eng. Sys. Safety 210 (2021), p. 107499. doi: 10.1016/j.ress.2021.107499 S. Tarantola, W. Becker, and D. Zeitz. A comparison of two sampling methods for global sensitivity analysis. Comput. Phys. Com. 183.5 (2012), pp. 1061–1072. doi: 10.1016/j.cpc.2011.12.015 S. Tezuka. Discrepancy between QMC and RQMC, II. Uniform Dist. Theory 6.1 (2011), pp. 57–64. url: https://pcwww.liv.ac.uk/~karpenk/JournalUDT/vol06/no1/5Tezuka11-1.pdf I. M. Sobol′. On the distribution of points in a cube and the approximate evaluation of integrals. USSR Comput. Math. Math. Phys. 7.4 (1967), pp. 86–112. doi: 10.1016/0041-5553(67)90144-9 I. M. Sobol′. Sensitivity estimates for nonlinear mathematical models. Math. Model. Comput. Exp 1.4 (1993), pp. 407–414

Comparing methods of randomizing Sobol′ sequences for improving uncertainty of metrics in variance-based global sensitivity estimation

This paper introduces an alternative way of randomizing Sobol′ sequences, called the Column Shift method, for reconstructing replicates to improve estimation of the uncertainty in sensitivity indices. The Column Shift method provides reliable results when applied to variance-based sensitivity analysis of the V-function, with much higher accuracy than commonly used randomization methods in most circumstances. It also addresses the error spikes caused by determinism within the Sobol′ sequence. The Column Shift method is compared with other popular randomization methods for the Sobol′ sequence, and it is shown to be the most consistent of those tested. In addition, the inclusion of standard error in the mean of sensitivity indices in an analysis of replicates provides a good indication of underestimation of errors in simulation results. The relationship between the number of samples and replicates is also discussed.Xifu Sun’s research was funded by a scholarship provided by the Mathematical Sciences Institute, Australia, and the Hilda John Bequest of the Australian National Universit

Differentiation between adrenal adenomas and nonadenomas using dynamic contrast-enhanced computed tomography.

This study was performed to evaluate the findings including the time density curve (TD curve), the relative percentage of enhancement washout (Washr) and the absolute percentage of enhancement washout (Washa) at dynamic contrast-enhanced computed tomography (DCE-CT) in 70 patients with 79 adrenal masses (including 44 adenomas and 35 nonadenomas) confirmed histopathologically and/or clinically. The results demonstrated that the TD curves of adrenal masses were classified into 5 types, and the type distribution of the TD curves was significantly different between adenomas and nonadenomas. Types A and C were characteristic of adenomas, whereas types B, D and E were features of nonadenomas. The sensitivity, specificity and accuracy for the diagnosis of adenoma based on the TD curves were 93%, 80% and 87%, respectively. Furthermore, when myelolipomas were excluded, the specificity and accuracy for adenoma were 90% and 92%, respectively. The Washr and the Washa values for the adenomas were higher than those for the nonadenomas. The diagnostic efficiency for adenoma was highest at 7-min delay time at DCE-CT; Washr was more efficient than Washa. Washr ≥34% and Washa ≥43% were both suggestive of adenomas and, on the contrary, suspicious of nonadenomas. The sensitivity, specificity and accuracy for the diagnosis of adenoma were 84%, 77% and 81%, respectively. When myelolipomas were precluded, the diagnostic specificity and accuracy were 87% and 85%, respectively. Therefore, DCE-CT aids in characterization of adrenal tumors, especially for lipid-poor adenomas which can be correctly categorized on the basis of TD curve combined with the percentage of enhancement washout

Tumoral angiogenesis in both adrenal adenomas and nonadenomas: a promising computed tomography biomarker for diagnosis

To explore the correlation between the typical findings of dynamic contrast-enhanced computed tomography (DCE-CT) and tumoral angiogenesis (microvessel density [MVD] and vascular endothelial growth factor [VEGF]) in adenomas and nonadenomas such that the enhancement mechanism of DCE-CT in adrenal masses can be explained more precisely. Forty-two patients with 46 adrenal masses confirmed by surgery and pathology were included in the study; these masses included 23 adenomas, 18 nonadenomas, and 5 hyperplastic nodules. The findings of DCE-CT and angiogenesis in adrenal masses were studied. The features of DCE-CT in adenomas and nonadenomas were evaluated to determine whether the characteristics of DCE-CT in adrenal masses were closely correlated with tumoral angiogenesis. Adrenal adenomas were significantly different from nonadenomas in the time density curve and the mean percentage of enhancement washout at the 7-minute delay time in DCE-CT. The mean MVD and VEGF expression exhibited significant differences between the rapid washout group (types A and C) and the slow washout group (types B, D, and E) and between the relative washout (Washr) ≥34% and the absolute washout (Washa) ≥43% on the 7-minute enhanced CT scans (P=0.000). Adenomas were suggested when adrenal masses presented as types A and C, and/or the Washr ≥34%, and/or the Washa ≥43%, and the opposite was suggested for nonadenomas. These results showed a close correlation between the characteristics of DCE-CT and both MVD and VEGF expression in adrenal masses. There was also a significant difference in MVD and VEGF expression between adenomas and nonadenomas. In conclusion, MVD and VEGF expression are two important pathological factors that play important roles in the characterization of DCE-CT in adrenal masses because they cause different time density curve types, the Washr and the Washa for adrenal adenomas and nonadenomas

Tumoral angiogenesis in both adrenal adenomas and nonadenomas: a promising computed tomography biomarker for diagnosis.

To explore the correlation between the typical findings of dynamic contrast-enhanced computed tomography (DCE-CT) and tumoral angiogenesis (microvessel density [MVD] and vascular endothelial growth factor [VEGF]) in adenomas and nonadenomas such that the enhancement mechanism of DCE-CT in adrenal masses can be explained more precisely. Forty-two patients with 46 adrenal masses confirmed by surgery and pathology were included in the study; these masses included 23 adenomas, 18 nonadenomas, and 5 hyperplastic nodules. The findings of DCE-CT and angiogenesis in adrenal masses were studied. The features of DCE-CT in adenomas and nonadenomas were evaluated to determine whether the characteristics of DCE-CT in adrenal masses were closely correlated with tumoral angiogenesis. Adrenal adenomas were significantly different from nonadenomas in the time density curve and the mean percentage of enhancement washout at the 7-minute delay time in DCE-CT. The mean MVD and VEGF expression exhibited significant differences between the rapid washout group (types A and C) and the slow washout group (types B, D, and E) and between the relative washout (Washr) ≥34% and the absolute washout (Washa) ≥43% on the 7-minute enhanced CT scans (P=0.000). Adenomas were suggested when adrenal masses presented as types A and C, and/or the Washr ≥34%, and/or the Washa ≥43%, and the opposite was suggested for nonadenomas. These results showed a close correlation between the characteristics of DCE-CT and both MVD and VEGF expression in adrenal masses. There was also a significant difference in MVD and VEGF expression between adenomas and nonadenomas. In conclusion, MVD and VEGF expression are two important pathological factors that play important roles in the characterization of DCE-CT in adrenal masses because they cause different time density curve types, the Washr and the Washa for adrenal adenomas and nonadenomas